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(54) TIME-SPACE INTEGRATION AND MANAGING METHOD FOR A PLURALITY OF VIDEO IMAGES ITS 
DEVICE AND RECORDING MEDIUM RECORDING ITS PROGRAM 

(57)Abstract: 

PROBLEM TO BE SOLVED: To provide an integration method and 
device for a plurality of video images to manage, express and 
operate them in unified way, in which video information is efficiently 
acquired from a plurality of video images in a same photographic 
space. 

SOLUTION: A photographic state detection section 103 reds an 
image data string to detect camera on/off information and camera 
operation information. A video split section 104 splits a video image 
into shots based on the camera on/off information, and an object 
background separate section 105 separates an object from a 
background based on the camera operation information. While an 
object motion information extract section 1 06 makes object 
information mutually correspond between frames, a photographic 
space re-compositing section 1 07 composites again the camera 
operation information and the background and an inter-shot relation 
calculation section 108 calculates a spatial relation among a 
plurality of photographic spaces composited again. The camera 
on/ off information, the camera operation information, and each 
information among the objects, the motion of object, re-composited 
background and shots are managed and stored in time space and 
one object or over and the image pickup space are composited 
again, according to a request of the user or the like and the result is 
displayed or outputted. 
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7ftf^flt«S:affl1-5„ 
[0 0 2 5] flfflJ^ffittfflgB 1 0 3 ©*J$£0 2 [Otc 
1% H 3 HN&RfHfttllfl! 1 0 3 © 2 0 1 ^5, 2 
0 4ST*©*^*t±5^a©»fctt^1- 0 iilbO 
Hffi&l^T, #«fiic*S©#^i:^a©^* 

[oo2 6] io2©H*7*-^ju^*ygija»e>i»^tfi 

S*lfc l*7**fll#0*7 U-Af±, 0 3 t/Tc-f 3 0 1 T? 

~©3oi£iit, #£raiifci:nf//rv5. r 

l., 2 o i iDmB^w-tau^^ 

1-5. Sff^ftSJIJItt}!}, 0 3"Cfl3 0 2T?*t), * 
¥^[Rlil^ffiii, 0 3T*f±3 0 3T'&5„ rftib©# 

^fflTltfe^fc 3 0 4 SrSi^iRiiitit&ric^sraiiift, 

3 o 5 *7k¥iifammm£?nm{ktm&, 

[0 0 2 7] ifjftvt, 2 0 2©if{|7*-*3fc'<#*.S|S 

atflix-csit, ^^frttiu^B^imifes o 4, 

3 o 5©&tf#*.£fT5. rr-e©MU«#x.^a[i, H 

3 3 0 6 Mm u m^(D&mtt?¥mx£f£r$ 
m<&%®m-fz>%mxhz>> yu-j±m&<Dx, y mm 

ttiSa?r^t*|RlSrlSffi©ft^[6]t LTV^5 C ®m 

^ If 3 y ©^iff T'ffl V ^ *IT V ^ 5 , * ^ 7 ©itfT^lR) 
iMffiw&ll^tf^ffiT', BfSnHff^DlfLfcB#© 
HK-ffi (^t°* 0 -7¥ffiH^ (Ep i p o 1 a r PI 
ane Image) ) #&5. r ©Bf S^ifBll^^ 
feffi¥ft©Hft7nf4g^lt^LTV>5 D r^li, r©x 
t °^-y5P[gjiift±T\ #>*©WR^©«U«;iSjli|ftij:ft 
5 , r ©jl^©fl# ^fr#^©i!)§ ©^:# § 

r tiCioTV^5 [R. C. Bolles.H. Bak 
er.and D. H. Marimont, "Epip 
olar— plane image analysi 
s : An approach to determin 
g structurefrom motion", I 
J CV, 1, 1, pp7-5 5, june 198 

9. ] . B#£Pflitfgj£x, ymmib&$tti:o\z®mL 
tzm^wmmitzmz x - 1 ^ishba t mm 

ty, t M^tt^^tf ^H§IBfii«Sr y - t tir£Wm 
ft k BP ^ fit© y ©fit^^ff «© x - t B#SfflBt^ 

Bf SP B Wt?iJ t Pf y - t Bf^p^iiff ?iJ >b 
5 0 

[0 0 2 8]!^ Btfl7*-^M'<#^SS2 0 2^*5 
V^tfllOrSiifcSE, *¥*I^i£itlM^SIfflHili©9I 
BfffiSr, 2 0 3©^'lfS7^/l/^-^ag|5(-*5V>T7 

-f/^- (lB-*ifc^, *~*f»^)i) ^a^Jtto r 
3r$ttti1-5ri£gKiiL-Ci^ 0 H3-CH:, 3 0 7I; 
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[oo2 9]fs^t\ 2 o 4 <Dt^ma^ts^x , m 10 
ypm&v&B-ftfaizMWt&mtZo m3x*ix 3os 

fctBSU 3 o 8 aoM^iBjtJqg&gtrfTSo -© 
JniMaatt, ft© 7 ^ a-* -Ma ir*sv s tifcft 

^n-^ii^iastoi^ 

SfctEttSftS. m «ft©#»^©B#iS, n- 

Op/hU HlJ|L«HllS*K:KII*S*ifc< <fc<5. Sfc, 20 
£ ©JP^MSil, n^&a 31 W >f xkisv * 

C (t) = s um (F (x, t 
+ s um (F (y , 

AM* 4 # * 7&ffr2> 7 **i 

fe *>e*ff $ ft t v > 5 „ f£ [d 1 j , 
74t/*x (*^9HS) . ^>4 0i (#;><7£fe& 

fcS5*fP) , X-A4 0 2 (Wlfc«fcS*5£J:fc: 

J: »/M-$iWfO , fvwh4 0 3 (# 
^fciTfcSSjftffO , b7y*4 0 4 (U/y*£ 30 
SWItSiff) , 7*-A4 0 5 (t^7£±T(-# 

1MB i-ftfcfe, 7^7**tt»Jh, Ay, r 

-f^MS, 7W^M©3frft?jft©'M;, X- 

&S£fls£#v\ h7^, 7*- A, Ky-ttWSft® 
=*7clBglffft^iJt^©»#©^(^^ff-e&s 0 h 

[0 0 3 2] ±fB©*fe-Cj|[W Stbfc x - t 

w^*f ( X) t) ttmto x - 1 m^m^mmmit, 

£m x £ t ©Mfrefc 5„ flfPsI 1 0 © x - 1 ^sra 

&I«©«^£F (x, to) T'SU HflHtl* 
Kt.Ox-tlSaOTiWBWftosSWatfF (x, ti) 

h^7^-^SrS5fet5. fiTF, r©7^7»^ 50 
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^ixSHHAStttrl^na^Btt^PF^. x - t Sit 
^foBt^IItffe^ f, ft, x - t R^K&J&Mft 3 0 9 

iiM&feJi, y - t ti.W-lifa^mm&.frb y - t B#SP H 1 
S^Hff 3 1 0 fls#P>;h,3. x - t Bf^gi>iHfe©B# 

yjft©»#£«SU y - t Bf^p B W^©^«ii±Tyj 
[ 0 0 3 0 ] fl^T, 2 0 5 ©ft&ffitHSftfgBKfc^ 

^WtSit^a^^tliLT, *^7©^-y^-7ttl8 
©tttliSrfT5 D Aftftfcfi, x-tfl#£P H TOPHfe£F 
(x, t) , y-t^WSKH«lSrF (y, t) -e« 

z-t~?${m±xhh\-£, zomm t -?%t7<D*>*7 

[0 0 3 1 ] 

[ftl] 

) ) dx 

t) ) d y - (1) 

[0 0 3 3] t> MftSfc* ftfc x - t Bt^palg^M^^^ 
n-zWiKi^a^SLTV^tl-ntf, F (x, 
t») tF (x, t.) ©IHtKTOBB<R^*>5. 

[0 0 3 4] 

[»2] F (xX, t.) =F (a x + b, to) 

y - 1 m^m^mmat, 

[»3] F (y%, ti) =F (a y + b, to) 

[0035] H5^©50iiix-t ^^IBft»BfftSr 

SU 5 0 2l«P B lT©£P f fl#*F (x, T) , 503 

mmT-KD^mF u, t-d zztizm 

to 5 0 4 ©MSHlfcBite^i-.k 5 I^JS#it^tTV\ 

^/Sffitl 5 0 5 ^^OJ-TSo r©Itt|*&©ffit, 
ft«H - 1 ©ffiBBBfllfcSrlFtil LftiSftttS r i fc bTIBt? 
hz> 0 r©*fJS#itbtifc 5 0 4 i 5 0 5 (i s ft*©ft 

ttHt^5 0 ^©^^©{gt^X-A/N 0 ^^-^ a £j| 
[0 0 3 6]^V^T, **5Wlfer<7/-#Xto¥mc 

<t 19 , ±BT?#j6#»t t>titz£m&wM&m->x% y 7 

ftf^<5^-*Sr*ai-5 0 Afr^lciti, EH5 0 6© 
a i b Srlfffli-Sfcftir, #J6#Jt fcJifcffilBjSaMt* 
ffl^TWT©l^Tv<7^-^^(c|t^ (ftK) l 
T*f^^^©*7Cfi 5 0 7 Irttffl L, y<7 * -9 a , b 
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5» ro«*ii, -jRKHough 

SI (P. V. C. Hough, "Method an 
d Means for Recognizing C 
omplex Patterns", U. S. Pate 
nt No. 306954, 1962] kW&tlX^Z 
<>©-?*> 5 „ *fJ£#rtfcixfctt*©J!aiSrxXt x-C* 
atbOBi«fiTfBS:©J;5l'ft5. 

[0 0 3 7] 

IiC 4] b =x% • c o s (a) + x • s i n (a) 

H o u ghllit -jfttaiRfflw^i^^iifeojSis 10 

*©ftH£g?U W«S^t*«:iaoft*©S^5 o 7 
©ffiSffliSttffli-^t j£iS©fi§ i3Jf£Sl LTV^ 0 

[0038] tumtftt, aaisr*f»sMtcftsL, s 

^©a^ftSrSH-ffiSffiSrto-cttffit^tESrofit 
i[©«-*SrltjKSMtftSIL-r^7P«-^Srim}LTV> 
[0 0 3 9] IH«ty -tBlSWKfBMft^&TVA'h 20 

oTt, ;^d« (SP5>«i) Jcii = 2fc5ciRf^*'d-* ftv^ 

tfciu&a^Brte-eks. &±^auiwsttttiA i o 
3 ©ff^t&awstve&s,, 

[00 40] HlfcR3, 1 0 4©§«i#fiJ^C 

^flNWClr^V^-ClMftSr^a -7 r© 30 

a v hrtroifftii, ii$eW&^i:£Pi©ff«£<boT^ 

[00 4 1] ftfc, 1 0 5 «, WJ^K^lC;^ 

0 5O«j*«rH6l^-r. Hi^Hl^i 

0 5© 6 0 l^fc6 0 44T"©*^gS©*^t^f, 

t J: 5&a©«ix£P»cl&9!-f3 0 

[0 0 4 2] 4f\ 6 0 l©*p«7ftf^fB«Sl5t*JV^ 
T> BWfe7 U — AaEj^^S{c«t *) . **7%Wffl£& 40 

tin?, mmt h w&f- mmz ivt 
ztcmt • miLfttirtm%.T~$mxmk • a&fits* 

5o PS1-5Hffr-^^F (x, y, t) , F (x, 
y, t + 1) h+*. *^9HkfPAI4 (AflvMJ^ 

[0 0 4 3] 

[$C5] F (x, y, t + 1) = AF (x, y, t) 

y*MJ\ ft©SCT«*5. 50 
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[0 0 4 4] 

[ft6] F (x, y, t) =A" F (x, y, t + 1) 
jf^T\ 6 0 2 ©Milr-^ Jtg^Tfi, H^ftyi^s 

o . ±.Bx%/7um^^y-t?i>£fotzmm-rz> 

^Ir^-tyir^LfcIiiff if 5 L©JtRfeaT*fe !9 , *ttj 
3Ji5t>0tt»ftlH©JWt £^©fif$R©g#©f&*Hl 

^ft©ft#©^©^#^Wi:i©jl#i LT»ffi$ft 

[0 0 4 5] fc^T, 6 0 3 ©JfcRMfeT^JfctMHS-e 

*o«yifcJ:D, a&omifctzm#.7 t -9\z&^x 

*©*A/*©iift©ft£ft mmmm ©ipn^ 

[ 0 0 4 6 ] m^X, 6 0 4 ©Miti^Tfi, £1\ 

ffiw £*ifc«££#g#© 2 2 tt^a^st 

2{ift^a^tT5o Jfcfex-^Mfc&f (x, y ) t 
U 2imm®%F (x, y) 

[0 0 4 7] 

[«7l 

F (x, y) =1 : if f ( x , y) 

F (x, y) =0 : if f (x, y) <S 

B 7 £ 2 ftft^a©f!) fc**-. 7 0 1 # JfclfcT - * Hit 

-?*>•>, 702 a* 2f[fts*-e*>5 0 mm 9 1 

[004 8] fftvt\ 7^#!t#SicJ;^ 2»ffl 
^F (x, y) (Dy^V^tj^fsOo 7^#it©7^ 
3yXA^|8l^t P 2«kpS«lF (x, y) £F = 

(Fu } , 7^</Hi«lSrL= {lu } T-*to lu 
{i#^ilt^©7^Sr^-f-iESm-efe5 0 £fc s lfi 
illM^S^SMit T (i) tt^^/Hi**^ 
ix*1- 0 8 0 1 ©lOJHRSKlfc^T 1 = 1 1 U M* 

(2, 2) ^b7-<^SSrl^-r5„ g£©W3fS£ 

(i, j) iU 8 0 2lCjo^TW»f't5o fi.j =1 
4tjtf8 0 3^ fu =0*Pjlu =0iLT807 
^ftaS:a*5. H9(^L^*g^T\ Sft©!!* 
x»= (i, j) t, ^©fi#T:«^^©H^^El 1 
Od^-f <£ 5 tSU x,07^ (HffL©{() 

(p = l, 2, 3, 4) t-fSo 8 0 3-Cfi, {T (1 
p) , l,&0Xfc\<\ p = 1, 2, 3, 4} fplcMft-S 
EOfttf n agfc Sit, ^tifc*/J>$V MP t> JI (C L 
i, L 2 , L n i-t"S„ n = 0!iy±"8 0 4^ n = 
I4?)tf8 0 5^ n= 2/ t £fciJ8 0 6^aSrii*, 
#^a^, 8 0 7-^x1^, 8 0 7t'ii^ii*lc:o^Tll 
TLfc^if 5^W»fU ±H*»T-C&n«8 0 8, 

8 o 9©&a£LT7-<y >?f:%T+& 0 
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[0 0 4 9] &^T\ HMIitifgU^, y-< 

tx\ nxmtftmtZo 

[0 0 5 0] &±.frmt$, ffftftHK? 1 0 5 cD*J$i 

tmomti-zbs. io 

[00 5 1JRIC, 01tg5, 1 0 6 ©gt^ftfttfit 
StttfilHSKiS^-C, JH-g-#atJ;!9, ^7U-AIfS 

[0 0 5 2]-*, 107 (Dmim^m^m^^ 
^m&i^r^i^i^ t?», ira^Km 20 

0 5-e*»3ftfctf*S:* ii^lMttif! 1 0 3 -c*(±j 
A*riH*.5ii:v«ig|Bi: LT#£$Lft*s-r o mil 

^ff-p-j^©^ir*i-o Hi (a) m^t, iio 

1 1 1 1 0 2 lil$fflWfc:*tt*37 ^-AMft-cfc "9 , 

/>y(D*^7^T«$tLfc^-C'fc5^ 1 1 0 
2{il 1 0 1\ZMLX1 1 0 3 (7U-Afefci9(»/Ny 

#& 1 1 0 4 (7 f-A&fc "9 ©T ^ A- Mftf^*) 30 
17^7 hLT-a-^t^o X-Ajftf^(0#^, 01 1 
(b) t/Tti5l:, X-AJIl 1 0 5W&CTHffe<0 

7-7Srffl^o#^co^^o 0 ro^ii, ffi{g>£ 

[0 0 5 3] 1 0 8 ©->h y b Wl^ffif|5T% 

IH^M^St 4. <9 , ff£j&£tlfc^>3 y bfe© 

mm l < 4 s i 5 ks^ib Lfc«, -> 3 ^ 40 

3 y b©Srai-H1-5lta-C&5„ li«#rtSrfr5^a 
y btE®$ti-C^3^timo^O|BSP#?rv'3 
y MHS«flmt1-5. roi/a y Mffl©B8ffitt, 
SW^Wl 9 7-eS-a-^Lfc/V7v^H^Jtt55t5 

3o H 1 2 fcnttXHoS^SrSrf. =ft%£F B 1±©il 
120 1(1, RttaSMciDHttl 2 0 3±©1 2 0 2 
^**£*l5. 01 2f>©,6;Oi4, i7^7©Sj^if^T* 50 
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*u, Rtti 2 0 3±i±, M«ffit?*)5. ne©*^ 
W\ «^Bfwl^»f ©*t§-e-*T»*s<5 0 r© 

b i: LTffiZHSmffc&J^T^yT^KJ: 9* 
1ft "9 fcffNBHSfcSriffc t^itt 5 r i 

v y TWtf we&a. 5ii4feiw©ffi 

fftfttfcfi, PM^£%JtLT^y^^Lft(7tU4 
ftftV\, -o©M#*S*i LTl^Sglf f ^>L-5o 

^ft $ *r v y f- y j/f 5 r i r« *g ttj-r 3 r i # 

7?i± s B^OTct $S-t£± • ffi/h^*Tvyf-^^-f§ 

[0054] w-mztitcij/yj-ytymn, 11*7% 
&m, mmn, mmzm, w&f&Lf-m 

iwt*3gttata • 1 0 9 \mmmm^ m ■ w 

tt r g b (Dim t Lxwmmmn $ a s 

vm4, tt*ofc*r*tflH5-©«ia, m m^-tz^t 

SS. Lf St" 3 r t "C\ RGB ©R£|Hjafls{S#©3fc-e* 

mmtt urn 5 ^©h^i, ^wi-M-t-5«*^ft* 
f B iwd« • t mtb^txxmict^xum^x^ 
[0055] m^htz^mmmn, wtf-tK. 

^7^-7©^ft^ 7-^ffl^^M7-^^L 

fcssflr«rnsi-5o &wmm(Dqx'tiTmmwtk 

J PEGff-^-ftTfcSo 3 20X240Cfft-e*>7 
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m¥ i o • 

24 



16 4 4 3 5 



£3 6 0«*spK:|lltTJHU$tifclMfc*»e> % #3SBJ1£ 
U J PEG«F-W:*fflv>5i i o #© l ©ESSMsas 

^i±, IU«i'#^ii^b-Cfe5H2 6 l^MPEG 
^©flFMl&fflV^fc-ejl&l 0^Oli>5)2 0^Ol 10 

ftt»?*SrgS!Lfc^hn#iift^) i LT JPEG 
JHt^fclTfftfitS^©, a.-if-WgjftfcSitSfc* 

IH®3b*fc i: -f y# 7 ^ h y ^ o A * tssa 

[0 0 5 6] 1101 ©S^^Tii, 112® 20 

w*i^ot, Rftffitw «t a • mm 1 o 9 a 

©^©ifctft if fcffr* WC*>5 a «fM t ^©Bfefctif 
«4»fctt«ttf«4rffillI^HHLieiBfl!)*oflMli LTit 

5 ft/V 7VIIS Sftfc^MH&fftSr* h p jJ?«I^ 

fc<5o 

[0 0 5 7] 0 1 3 (Clf^ ^IBlifWy y;?'©7 hP 
jKSS©^*^. 13 0 1 ^s-g-^^tife/V 5 v 
aiBTfcS. 1 3 0 2^S?ftSrSLTV^5 0 1 3 0 3 
«sag|Rfi«j-J-y^5y^Snfc^ hn#*aT?*>D, 1 3 
o 4^B#f H i^i'-y-y7 B y y^snfc^ hp#«3l-c* 

5. 1 3 0 4(1, S^#0K«#ft¥ft©S£'t)g?IL 
T*JD, 1 3 0 311, tfc^froa&nOmaaitSSE'ft 40 

SraSSLTV^. *ayh|«IHftflHRfcfli^"C, lift 5 

^3 -7 ht#S1-5lii[©fe¥fr*-0©ti;{C^1- 

U y hBtf|f££ixT^5S^fr (S¥B) £x 
hvtfm$L±.^*~-;<~-y y^LXWmt LX^TfrfZ 

^htt, f^jStfJ, T¥ttm&k<D7*- i»©a^ftif 



[0 0 5 8] 1 1 0 Til, a-if 1 1 2 

ifoK*fc*i:TS4?*ft*Si s Rrffitft 

tt\ 1 1 2©^-1fA*fl5tr^LTa-f ! ©g*t^t 
5^S©7^-K^y^^BTH^fe5 0 113© 

5 r t ttft < . *38K©5iSr5&«iLftv^HlJ:*iV>-r 
ffi* ©SEHas BTI6-t?&5 r i (iff 5 £T*fc ftv\ 

[0 0 6 0] 

mW(D%m W±©i5^ **HfcJ:*i.tf, BSMfev* 



Slifc1Wftoill5ffilWolWoailIttl^>3 y 

tz<DX\ l»ft©B#^raWft^^y7^y h, Bf$ 

[0 0 6 1] t1t y ±ISf a • *i«$^-CV^ttf«*S 

5i5i-Lfc©, ^C^ffi^^L^iSic©S*^t»L 
X, ^-^^a^g^^cT^-S©^^^/^!! 
i*^©'tf J: < 5 r t i»nT 

[HffiWffi^ftlftBJ] 

[01] *«P^©-HM«©*^i^a©StL^ 

[0 2 ] ±faHW^«^*3(t5«f^B^fflSP©*jt 

[03] ±E^»»Jic:^(t5l^l^yifp©Ma 
©sSti0T'fc5 o 

[04 ] tmmmmm^&tfz** ymimmzmw 
tzmxhz, 

[0 6] mmfmimzmzmfc, m&m&n 
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WmW-1 0- 1 6 4 4 3 5 
26 



[07] ±&mmi&M t& it § 2 ft{b^acD« sr^-r 1 1 1 

mvh%o 1 1 2 

[0 8 ] ±mmMMMm\z-&ttz> y yynfmom 113 

tiia-efc5 0 114 

m 9 ] ±iB^w^i^t*5(t5Mtesii^*i-ia-e 2 0 1 

20 2 

[010] ±&mmmi it 5 ^hss^s^ 203 

^•M^^-flil-efcSo 204 

[011] ±|ElU6^ffi^HfcMt«*^ 7tI££K~3< 10 2 0 5 

a*SW©ff^^£SrBill1-50T*>5 a 3 0 1 

[012] Rttf*^tait5@tJ,5o 302 

[013] mmmmm^ i 5 am ^i-y-^ y >- 303 

vx2#%M*mm-fz>mx*hz> a 304 

[«F*©RW] 3 0 5 

1 0 1 t-T^ff-^-A^SB 3 0 6 

102 Bffer-^flM^ytfl 30 7 

1 0 3 ftgRttttttfll 3 0 8 

1 0 4 mt^mU 3 0 9 

1 0 5 m-fo, HJkftim 20 3 1 0 

1 0 6 w&imzmttim e 0 1 

1 0 7 tf^ffl^ScfG 6 0 2 

108 is a y MIQHffilFttlAS 603 

109 BftftffiiH* tttf a • * 6 04 



111] 



[02] 



[06] 



-102 



iM&KSHBfflgs -103 



1 



-104 



-105 



106- «?fla»§«*ttaa 



114 



^107 



108 



no- 



T 



_L 



-111 



-113 



r 



[ 



3 



X 



I »«»«7-f^^-ffi ^ 203 



}~204 
]~205 



701 

0 12 0 0 0 10 0 

0 0 6 25 10 0 0 

10 0 20 30 45 10 10 

20 10 25 0 0 3 10 

61 0 0 0 0 10 SO 

0 0 0 0 0 0 25 



201 

}~202 I mtor-rttim 



X 



X 



strnmr-jr 



[07] 



2mt 

W«=9 



x 



-601 
-602 
-603 
-604 



702 

0 
0 
1 
1 
1 
1 




[Ell] 




(a) 



(b) 
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